Tank cache errors and timing

Tech Note: TN0963
Product: OpenEx, Synapse
Version: All
Date Added: 2015-09-09

Issue

OpenEx or Synapse stops recording and displays a "Trouble saving data to Tank" or "Tank unable to cache events" error message.

Solution

Errors related to data storage and timing can occur when saving high rates of data, your PC's hard drive is busy or full, or communication between your PC and your TDT system fails. To narrow the possibilities, look at the timestamp comparison located at the end of the error line in the messages window.

A timestamp comparison of:

  1. "-6.x vs 6" tells you that OpenWorkbench/Synapse can't store the data fast enough.

    1. If you don't need access to the tank data right away, increase the Cache Delay to the maximum (30 seconds). In OpenWorkbench, this option is found in File → Preferences menu. In Synapse, this option is found in Menu → Preferences → Advanced.

    2. Try increasing the Target Bus Loading setting to ~80%. This puts more resources towards storing the data and less on distributing it to applications that are viewing the data, like OpenController or the Synapse run-time interface. In OpenWorkbench, this setting is in the Setup → Polling and Performance menu. In Synapse, this setting is Menu → Preferences → Run-time Ops → Data Read Limit.

  2. "-18.x vs 6" (or similar, where the first number is ~3x larger than the second number) tells you there is a problem saving data to disk. This may be because:

    1. The hard drive is close to capacity or something else, like a virus scanner, is demanding heavy access.

    2. Check your hard drive storage capacity. If that is not the problem, try turning off auto virus scans, update, or other auto-process including Superfetch (aka prefetch) and Indexing. See Tech Note #0237 for more information on optimizing to avoid errors.

    3. The hard drive is failing.

      1. Tool to test hard drive can be downloaded here:

        https://crystalmark.info/download/index-e.html

        https://crystalmark.info/redirect.php?product=CrystalDiskInfo

      2. Note the following 5 best Self-Monitoring, Analysis and Reporting Technology (SMART) hard drive stats that indicate impending disk drive failure as reported by Backblaze:

        SMART 5 - Reallocated_Sector_Count
        SMART 187 - Reported_Uncorrectable_Errors SMART 188 - Command_Timeout
        SMART 197 - Current_Pending_Sector_Count
        SMART 198 - Offline_Uncorrectable

        SMART is a monitoring system included in computer hard disk drives (HDDs) and solid-state drives (SSDs) that detects and reports on indicators of drive reliability, with the intent of enabling the anticipation of predictable (40%) hardware failures.

  3. Any other timestamp errors:

    1. Make sure there is regular data saving to the tank. The tank server needs regular data storage or else it will stop responding. If there are no stream stores in the experiment, or only stream stores in 'View Only' mode, then you might see this error if there are only sparse epoc or snippet events occurring. Enable the Tick Store in the Synapse Processor Main Tab.

    2. Check for communication failure. Try reseating the fiber optic cables on both the RZ and PC sides. If errors continue to occur, contact Support.